Skip to content

fix(pool): cap accounts per request, scope bare 429 by evidence, keep failed hops in debug log - #1127

Closed
panjinhe wants to merge 1 commit into
dofastted:mainfrom
panjinhe:fix/failover-hop-cap
Closed

panjinhe wants to merge 1 commit into
dofastted:mainfrom
panjinhe:fix/failover-hop-cap

Conversation

@panjinhe

Copy link
Copy Markdown
Contributor

Problem

One auto-mode classifier request (sonnet, ~140 KB system prompt) got a bare 429 (no anthropic-ratelimit-* headers, no usable body) on 8 accounts in a row:

  • Every bare 429 was classified rate_limited_unknown and cooled its whole account for 60 s, so one request took 8 accounts out of the pool. The pool went pool_rate_limited eligible=0 for about a minute and other clients got 18 × 429.
  • max_account_switches / max_total_attempts were already set to 3, but they are not hard limits: once they are passed, untried units stay reachable, so the request kept walking accounts.
  • The upstream 429 text was never stored. Docker logs are gone after a recreate, and classifier requests were excluded from debug bodies, so the real cause could not be found afterwards.

Fix

failover-runner.mjs (all new knobs live under failover in routing.json):

  • max_accounts_per_request (default 0 = unlimited, unchanged behavior). A hard stop after N accounts ran; the request answers with the last hop, so a bare 429 stays a 429.
  • bare_429_request_scope_after (default 2). A bare 429 on that many accounts of one request means the request itself is the trigger: stop with rate_limited_request and cool nobody.
  • bare_429_account_sessions (default 2) within bare_429_window_ms (default 120000). A single bare 429 only steps the account aside for that request. The account cools only once that many distinct sessions hit a bare 429 on it inside the window. 429s with rate-limit headers or a recognised body are handled exactly as before.
  • Failed hops (status, latency, rate-limit headers, body, provider facts) are kept per request (bounded map, 2000 entries) and written to the debug record as hop_errors.

request-log.mjs: classifier requests now keep inbound/outbound bodies in debug mode.

Behavior change

With the defaults, one bare 429 no longer cools an account for 60 s by itself. Set bare_429_account_sessions: 1 to keep the old behavior.

Tests

  • failover-runner.test.mjs: new cases for request-scope stop on two accounts, cooling only on a second session, the max_accounts_per_request hard stop, and kept hop errors.
  • request-log.test.mjs: classifier bodies are kept, and hop_errors is in the debug record.
  • Full unit suite passes on this branch.

Verified in production

Running since 2026-11-16 on 1.4.112 with these files mounted and max_accounts_per_request: 3.

… failed hops and classifier bodies in debug log

08:50 incident: one auto-mode classifier request got a bare 429 on 8 accounts in a row;
each 429 cooled its whole account for 60s and the pool went eligible=0.

- failover.max_accounts_per_request (0 = unlimited): hard stop after N accounts ran
- bare 429 on bare_429_request_scope_after (2) accounts of one request -> request-scope
  stop, nobody cooled
- a bare 429 cools the account only after bare_429_account_sessions (2) distinct
  sessions hit it within bare_429_window_ms; otherwise it only steps aside for the request
- failed hops (status, rate-limit headers, body, provider facts) go into the debug
  record as hop_errors
- classifier requests keep inbound/outbound bodies in debug mode
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T12:05:50.263292Z fd26062 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd260624e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +137 to +138
body: clipText(result?.body),
provider_result: clipText(result?.providerResult, 2000),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Redact credential-shaped strings before storing hop errors

When an upstream or CLI error contains a bearer token or sk-ant-... credential, clipText only truncates it, and RequestLogStore.finish persists the resulting hop_errors verbatim. Unlike the existing clampBody and debugResponse paths, neither this helper nor the new persistence path applies redactSecrets. Consequently, even caller-forced debug mode can leave usable credentials in SQLite and the panel's debug records. Apply the existing secret masking to the hop body and provider facts before retaining them.

Useful? React with 👍 / 👎.

Comment on lines +1258 to +1260
if (this.accountCapReached(ranAccounts)) {
releaseStickySlots()
return stopAtAccountCap(result, policy, ranAccounts.size, budget.hops, attribution())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the account cap on retry continuations too

With max_accounts_per_request: 1 and two available accounts, an uncommitted empty/incomplete hop takes the preceding allowSameUnit continuation before reaching this guard. That continuation sets avoid to the first account, so the next selection deliberately prefers another account and executes the request on two distinct accounts despite the cap of one. The exception path has the same ordering. Enforce the cap when selecting the next account, including retry continuations, rather than only in the ordinary switch path.

Useful? React with 👍 / 👎.

Comment on lines +644 to +645
hits.push({ key: String(key || ''), at: now })
this.bare429Hits.set(accountId, hits)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Discard cooldown evidence from request-scoped 429s

With the defaults, let one session receive bare 429s from accounts A and B, then let a second session receive the same responses within 120 seconds. Both requests end as rate_limited_request, but the first request leaves a hit for A in this map. The second request's first hop therefore triggers applyCooldown(A) before B produces the request-scope verdict, and neither the hit nor the cooldown is rolled back. Repeated request-scoped failures consequently still remove accounts from the pool despite the intended "cool nobody" behavior. Defer or discard account cooldown evidence for requests that receive a request-scope verdict.

Useful? React with 👍 / 👎.

@dofastted

Copy link
Copy Markdown
Owner

感谢贡献。替代修复 #1165 已合入 main(b1842a588e)。实际执行账号可设置硬上限;重复裸 429 只结束本次请求,保留有证据的冷却与额度探测。不保存原始 hop / 分类器正文,不使用跨请求猜测。 此 PR 不再重复合并。

@dofastted dofastted closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants